Algorithms and scheduling techniques for clusters and grids

ing the Lyon platform. [email protected] February 8, 2008 Algorithms and scheduling techniques 65/ 134 Introduction Parallel algorithms Scheduling Pipeline workflows Models and real life Conclusion Results First heuristic building the ring without taking link sharing into account Second heuristic taking link sharing into account (and with quadratic programming) Ratio Dc/Dw H1 H2 Gain 0.64 0.008738 (1) 0.008738 (1) 0% 0.064 0.018837 (13) 0.006639 (14) 64.75% 0.0064 0.003819 (13) 0.001975 (14) 48.28% Ratio Dc/Dw H1 H2 Gain 0.64 0.005825 (1) 0.005825 (1) 0 % 0.064 0.027919 (8) 0.004865 (6) 82.57% 0.0064 0.007218 (13) 0.001608 (8) 77.72% Table: Tstep/Dw for each heuristic on the Lyon and Strasbourg platforms ( numbers in parentheses show size of solution rings) [email protected] February 8, 2008 Algorithms and scheduling techniques 66/ 134 Introduction Parallel algorithms Scheduling Pipeline workflows Models and real life Conclusion And with non dedicated platforms? Available processing power of each processor changes over time Available bandwidth of each communication link changes over time ⇒ Need to reconsider current allocation ⇒ Introduce (dynamic) redistribution algorithms [email protected] February 8, 2008 Algorithms and scheduling techniques 67/ 134 Introduction Parallel algorithms Scheduling Pipeline workflows Models and real life Conclusion A possible approach If actual performance “too much”different from expected characteristics when building solution Actual criterion defining “too much” ? If actual performance “very” different compute a new ring redistribute data from old ring to new one Actual criterion defining “very” ? Cost of the redistribution ? If the actual performance is “a little” different compute new load-balancing in existing ring redistribute data in existing ring How to efficiently do the redistribution ? [email protected] February 8, 2008 Algorithms and scheduling techniques 68/ 134 Introduction Parallel algorithms Scheduling Pipeline workflows Models and real life Conclusion Load-balancing Principle: ring is modified only if this is profitable Tstep: length of an iteration before load-balancing T ′ step: length of an iteration after load-balancing Tredistribution: redistribution cost niter: number of remaining iterations Condition: Tredistribution + niter × T ′ step ≤ niter × Tstep Redistribution algorithms for homo/hetero uni/bi-dir rings (Well, let’s do this another time . . . ) [email protected] February 8, 2008 Algorithms and scheduling techniques 69/ 134 Introduction Parallel algorithms Scheduling Pipeline workflows Models and real life Conclusion Lesson learnt? Realistic networks models: mandatory but less tractable . . . Need find good trade-offs. Would be even more complicated with hierarchical architectures / [email protected] February 8, 2008 Algorithms and scheduling techniques 70/ 134 Introduction Parallel algorithms Scheduling Pipeline workflows Models and real life Conclusion Traditional scheduling – Framework Application = DAG G = (T ,E ,w) T = set of tasks E = dependence constraints w(T ) = computational cost of task T (execution time) c(T ,T ′) = communication cost (data sent from T to T ′) Platform Set of p identical processors Schedule σ(T ) = date to begin execution of task T alloc(T ) = processor assigned to it [email protected] February 8, 2008 Algorithms and scheduling techniques 73/ 134 Introduction Parallel algorithms Scheduling Pipeline workflows Models and real life Conclusion Traditional scheduling – Constraints w(T’) time w(T) comm(T,T’) σ(T ) + w(T ) σ(T ) T T’ σ(T ′) Data dependences If (T ,T ′) ∈ E then if alloc(T ) = alloc(T ′) then σ(T ) + w(T ) ≤ σ(T ′) if alloc(T ) 6= alloc(T ′) then σ(T ) +w(T ) + c(T ,T ′) ≤ σ(T ′) Resource constraints alloc(T ) = alloc(T ′) ⇒ (σ(T ) + w(T ) ≤ σ(T ′)) or (σ(T ′) + w(T ′) ≤ σ(T )) [email protected] February 8, 2008 Algorithms and scheduling techniques 74/ 134 Introduction Parallel algorithms Scheduling Pipeline workflows Models and real life Conclusion Traditional scheduling – Objective functions Makespan or total execution time MS(σ) = max T∈T (σ(T ) + w(T )) Other classical objectives: Sum of completion times With release dates: max flow (response time), or sum flow Fairness oriented: max stretch, or sum stretch [email protected] February 8, 2008 Algorithms and scheduling techniques 75/ 134 Introduction Parallel algorithms Scheduling Pipeline workflows Models and real life Conclusion Traditional scheduling – About the model Simple but OK for computational resources No CPU sharing, even in models with preemption At most one task running per processor at any time-step Very crude for network resources Unlimited number of simultaneous sends/receives per processor No contention → unbounded bandwidth on any link Fully connected interconnection graph (clique) In fact, model assumes infinite network capacity [email protected] February 8, 2008 Algorithms and scheduling techniques 76/ 134 Introduction Parallel algorithms Scheduling Pipeline workflows Models and real life Conclusion Makespan minimization NP-hardness Pb(p) NP-complete for independent tasks and no communications (E = ∅, p = 2 and c = 0) Pb(p) NP-complete for UET-UCT graphs (w = c = 1) Approximation algorithms Without communications, list scheduling is a (2 − 1 p )-approximation With communications, result extends to coarse-grain graphs With communications, no λ-approximation in general [email protected] February 8, 2008 Algorithms and scheduling techniques 77/ 134 Introduction Parallel algorithms Scheduling Pipeline workflows Models and real life Conclusion List scheduling – Without communications Initialization: Compute priority level of all tasks Priority queue = list of free tasks (tasks without predecessors) sorted by priority While there remain tasks to execute: Add new free tasks, if any, to the queue. If there are q available processors and r tasks in the queue, remove first min(q, r) tasks from the queue and execute them Priority level Use critical path: longest path from the task to an exit node Computed recursively by a bottom-up traversal of the graph [email protected] February 8, 2008 Algorithms and scheduling techniques 78/ 134 Introduction Parallel algorithms Scheduling Pipeline workflows Models and real life Conclusion List scheduling – With communications (1/2) Priority level Use pessimistic critical path: include all edge costs in the weight Computed recursively by a bottom-up traversal of the graph MCP Modified Critical Path Assign free task with highest priority to best processor Best processor = finishes execution first, given already taken scheduling decisions Free tasks may not be ready for execution (communication delays) May explore inserting the task in empty slots of schedule Complexity O(|V | log |V | + (|E | + |V |)p) [email protected] February 8, 2008 Algorithms and scheduling techniques 79/ 134 Introduction Parallel algorithms Scheduling Pipeline workflows Models and real life Conclusion List scheduling – With communications (2/2) EFT Earliest Finish Time Dynamically recompute priorities of free tasks Select free task that finishes execution first (on best processor), given already taken scheduling decisions Higher complexity O(|V |p) May miss “urgent” tasks on critical path Other approaches Two-step: clustering + load balancing DSC Dominant Sequence Clustering O((|V | + |E |) log |V |) LLB List-based Load Balancing O(C logC + |V |) (C number of clusters generated by DSC) Low-cost: FCP Fast Critical Path Maintain constant-size sorted list of free tasks: Best processor = first idle or the one sending last message Low complexity O(|V | log p + |E |) [email protected] February 8, 2008 Algorithms and scheduling techniques 80/ 134 Introduction Parallel algorithms Scheduling Pipeline workflows Models and real life Conclusion Extending the model to heterogeneous clusters Task graph with n tasks T1, . . . ,Tn. Platform with p heterogeneous processors P1, . . . ,Pp. Computation costs: wiq = execution time of Ti on Pq wi = ∑p q=1 wiq p average execution time of Ti particular case: consistent tasks wiq = wi × γq Communication costs: data(i , j): data volume for edge eij : Ti → Tj vqr : communication time for unit-size message from Pq to Pr (zero if q = r) com(i , j , q, r) = data(i , j)× vqr communication time from Ti executed on Pq to Pj executed on Pr comij = data(i , j) × ∑ 1≤q,r≤p,q 6=r vqr p(p−1) average communication cost for edge eij : Ti → Tj [email protected] February 8, 2008 Algorithms and scheduling techniques 81/ 134 Introduction Parallel algorithms Scheduling Pipeline workflows Models and real life Conclusion Rewriting constraints Dependences For eij : Ti → Tj , q = alloc(Ti ) and r = alloc(Tj): σ(Ti ) + wiq + com(i , j , q, r) ≤ σ(Tj) Resources If q = alloc(Ti ) = alloc(Tj), then (σ(Ti ) + wiq ≤ σ(Tj)) or (σ(Tj) + wjq ≤ σ(Ti )) Makespan max 1≤i≤n ( σ(Ti ) + wi ,alloc(Ti ) ) [email protected] February 8, 2008 Algorithms and scheduling techniques 82/ 134 Introduction Parallel algorithms Scheduling Pipeline workflows Models and real life Conclusion HEFT: Heterogeneous Earliest Finish Time Priority level: rank(Ti ) = wi + max Tj∈Succ(Ti ) (comij + rank(Tj)), where Succ(T ) is the set of successors of T Recursive computation by bottom-up traversal of the graph Allocation For current task Ti , determine best processor Pq: minimize σ(Ti ) + wiq Enforce constraints related to communication costs Insertion scheduling: look for t = σ(Ti ) s.t. Pq is available during interval [t, t + wiq[ Complexity: same as MCP without/with insertion [email protected] February 8, 2008 Algorithms and scheduling techniques 83/ 134 Introduction Parallel algorithms Scheduling Pipeline workflows Models and real life Conclusion What’s wrong? , Nothing (still may need to map a DAG onto a platform!) / Absurd communication model: complicated: many parameters to instantiate while not realistic (clique + no contention) / Wrong metric: need to relax makespan minimization objective [email protected] February 8, 2008 Algorithms and scheduling techniques 84/ 134 Introduction Parallel algorithms Scheduling Pipeline workflows Models and real life Conclusion Problem